A new codebook model was proposed based on HSV color space to eliminate the effect of complex dynamic background in the moving object detection. The merits of this new model lie in three aspects: 1) HSV color space was introduced to effectively distinguish foreground and background for false targets removal; 2) a 4-tuple codeword was proposed for fast codebook training and small storage in comparison with the traditional 9-tuple codeword; 3) a new codebook learning and updating scheme was designed for easy and fast codebook training and detection. A global quantitative evaluation method named recall-precision curve was also proposed for the video sequence. Qualitative and quantitative experiments demonstrate that the proposed codebook model can effectively detect moving object under complex dynamic background.